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Abstract 

Pattern recognition is much more disseminated and fundamental endeavor than usually realized. Arising from a 
need to achieve compact, organized and abstract representations of the entities in our world, it underlies not only daily 
tasks, but also constitutes the basis of language, knowledge, and modeling. In this work we revise the concept of patter 
recognition, its objectives and challenges, and present some simple but effective means to perform this important task, 
namely the k— neighbors and k— means methods. 


‘Tutta la nostra conoscenza ha origine nelle nostre percezioni.’ 

Leonardo da Vinci. 

1 Introduction 

Plants are amazing natural entities. Representing one of 
the earliest forms of life, plants have been fundamental for 
the development and maintenance of a myriad of other liv¬ 
ing forms, including animals and ourselves. Early in life, 
we learn to identify plants and trees, and progressively 
develop an ability to distinguish their varying types, and 
to know the respective names. This recognition involves 
taking into account several respective features, including 
the overall size, shape and color of the plants, as well as 
their parts (e.g. leaves, fruits, seeds, roots, etc). Then, 
when we observe the leaves of a given plant, we recognize 
that they share many features and often can hardly be 
distinguished one another. Our interaction with plants 
provides one of the most compelling and complete exam¬ 
ple of the important activity known as pattern recognition , 
which extends to many other types of living beings, ob¬ 
jects, and even actions. 

Indeed, the action of pattern recognition is much more 
ubiquitous than often realized. We recognize not only 
plants, but virtually every entity that can have implica¬ 
tions to our lives. Each of the terms of our very lan¬ 
guage, for instance, entails its respective pattern recogni¬ 
tion. This is self-evident in the case of substantives, but 
verbs and other types of words also involve recognition 


(actions, in the case of verbs). All other human abili¬ 
ties also relate to pattern recognition, including behavior, 
arts, and scientific modeling (e.g. HD- Indeed, prediction 
can also be understood as being intrinsically related to 
pattern recognition, in the sense that often we make fore¬ 
casts based on similarities with other previous instances of 
the phenomenon of interest. In brief, pattern recognition 
is at the very core of human activity. 

Being able to identify the category of a given entity has 
been centrally important for us, as it allows one to identify 
if a fruit is edible, if an animal is friendly, if an action 
is potentially dangerous, among many other possibilities. 
Thus, in a sense, pattern recognition plays a critical role 
for living beings. 

Figure [l] illustrates the basis of pattern recognition: 
given two objects, one needs to decide if they are of the 
same type or belong to different categories. 
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Figure 1: Given two objects, how can we decide if they are of the 
same type or belong to distinct categories? These issues pertain to 
the area of pattern recognition. 

The understanding of the term pattern recognition can 
proceed by taking into account each of its two parts: [pat¬ 
tern\ which basically corresponds to any entity/object of 
any nature; and ‘recognition\ a slightly more elaborate 
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concept that can actually be broken into two sub-parts: 
Ve ; , in the sense of repeating something, and ‘ cognition\ 
which can be interpreted simply as knowing. So, recogni¬ 
tion can be ultimately understood as knowing that some 
specific pattern has recurred. Therefore, cognition is a po¬ 
tential precondition to recognition. One important point 
here is that the recurrence of a pattern can take place be¬ 
cause the same object is presented again, or because two 
similar objects are presented. 

Another key related aspect in pattern recognition is 
that objects identification involves the definition of cate¬ 
gories (e.g. dogs, cats, horses, etc., in the case of verte¬ 
brates) into which the objects are grouped based on their 
relative similarity. This is ultimately necessary because 
we would be otherwise unable to label each possible en¬ 
tity in the real world. For instance, we would need to have 
names for all possible cats and dogs in this world, and so 
on. Abstracting similar entities into specific categories, 
while not undermining critical information characterizing 
the entities, can save a lot of memory and effort. 

The definition of categories, which is one of the im¬ 
portant and difficult steps in pattern recognition, should 
reflect not only the similarity between the considered ob¬ 
jects, but also take into account the importance of specific 
features. For instance, details that can have important 
consequences but are small, and would therefore imply 
respective little discriminability, need to be emphasized 
while defining the respective categories. 

Pattern recognition itself can be achieved in two 
main ways: (i) supervised , in the sense that exam¬ 
ples/prototypes respective to each category are available; 
and (ii) non-supervised, in which the categories of pat¬ 
terns have to be inferred from scratch. Needless to say, 
the latter tends to be much more difficult than the former. 
In addition, there can be no supervised pattern recogni¬ 
tion unless it is preceded by some unsupervised definition 
of categories. 

Effective and versatile pattern recognition has repre¬ 
sented a substantial challenge in science and technology, 
as a consequence of the lack of definitive means for select¬ 
ing features for objects characterization, as well as the fre¬ 
quent existence of overlap between categories. It is there¬ 
fore hardly surprising that the recent area of deep learning 
(e.g. [2]) has received so much attention. Here, massive 
data and computing resources, allied with new conceptual 
insights, is finally paving the way to effective automated 
pattern recognition as required by a wide range of ap¬ 
plications and even theoretical developments. More and 
more most activities will involve or critically depend on 
automated pattern recognition, so that it is important to 
be acquainted with related concepts and methods. 

In the present work, we develop a conceptual intro¬ 
duction to some of the main concepts and methods of 


pattern recognition. We start by discussing the neces¬ 
sary mapping from each entity/object into a respective 
mathematical, quantitative representation, giving rise to 
the concept of feature vectors. Because this allows one 
to derive models of the mapped object, allowing its un¬ 
derstanding, here we call this preliminary state pattern 
cognition. The next stage in which more than one pat¬ 
terns are compared, grouped and assigned to categories 
constitutes the pattern recognition proper, which is dis¬ 
cussed subsequently. Then, we introduce two of the main 
types of pattern recognition approaches: supervised, and 
unsupervised, and briefly present and illustrate simple re¬ 
spective methods, namely k— neighbors and k— means. 

2 Pattern Cognition 

Often, the entities to be recognized belong to the real- 
world, including birds, flowers, bikes, pencils, etc. Be¬ 
cause the respective recognition takes place in our nervous 
system (biological pattern recognition), or in computer 
circuits (automated patter recognition), it is necessary to 
transfer quantitative information about the object to be 
recognized into these two types of systems. This can be 
achieved by the use of biological senses or artificial sen¬ 
sors and equipments (e.g. cameras, eyes, scales, rulers, 
voltmeters, etc.), which can operate on more than one 
modality (e.g. image, sound, smell, etc.). 

These primary representation received by the pattern 
recognition system typically involve a large amount of 
data (e.g. an image containing millions of pixels), much 
of which tends to be redundant (e.g. the pixels associated 
to a limpid blue sky). It is therefore interesting to derive 
a new representation involving a smaller set of data but 
which retains as much as possible of the relevant infor¬ 
mation. The set of derived features or measurements fi, 
i = 1,2,..., M, can be organized into a respective feature 
vector f. Observe that data is not the same as informa¬ 
tion , the latter being more important. 

Figure [2] illustrates the mapping of the scene of a real- 
world object (a sea snail) into a digital image in a com¬ 
puter, which is then transformed into a new, more com¬ 
pact and effective representation /. In this case, possi¬ 
ble features could include width, arc-length of the spiral, 
color, elongation, etc. 

These two mapping stages culminate in a representa¬ 
tion f of the original entity, containing respective relevant 
information. In a sense, the pattern recognition system 
can therefore be understood to contain knowledge, as pro¬ 
vided by the several measurements in /, about the real- 
world object, defining a respective feature space whose di¬ 
mension corresponds to the number of adopted features. 
It is them possible to develop a model of the object, pro- 
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Figure 2: In order to be handled by an animal or computer, an ob¬ 
ject from the real world needs to be measured into a discrete space. 
In this particular example, a digital camera is used to transform the 
real-world scene of a snail into an image by sampling its colors, in¬ 
tensities, and geometry. The obtained image representation usually 
involves a large amount of data and contains much redundancy, so 
that more compact representations, capable of keeping as much as 
possible the original information while removing redundancies, can 
be subsequently obtained. The final product is the feature vector 
/• 


viding some level of respective understanding. Thus, in a 
sense these mappings can be understood as a ‘ cognition ; 
stage, in which knowledge about some relevant properties 
of the object are derived. However, this system is as yet 
unable to identify recurrent presentations of this same ob¬ 
ject, as well as relationships between objects, i.e. it is not 
yet capable of pattern recognition. 

Though conceptually simple, the mapping of an object 
into the respective feature vector / is by no means a 
straightforward task. The key issue here is to derive a 
small set of features that would be capable of establish¬ 
ing a bijective relationship between the real-world objects 
and the respective representations /. Bijection is partic¬ 
ularly important as it allows the inverse mapping to be 
defined, therefore establishing a unique correspondence 
between the entities and their respective representations. 

If too few features are considered, it is likely that sev¬ 
eral objects will map into the same representation. For 
instance, in case we choose as feature the length and width 
of an object, several objects will map into the same rep¬ 
resentation /, implying a non-injective mapping, which is 
non-invertible. It may also happen that some of the con¬ 
sidered objects have not yet been mapped into respective 
feature vectors /, implying in a non-surjective mapping, 
which is also non-invertible. 

Invertibility is achieved if and only the map is injec¬ 
tive and surjective. Yet, if too many features are selected 
aimed at achieving an invertible mapping, the representa¬ 
tion will tend to become redundant and make it very dif¬ 
ficult to populate and handle the respective feature space 
with samples of objects. Regarding redundancy, it is in¬ 
teresting to observe that, given a relatively simple entity, 
there is not much that can be said about it without en¬ 
gaging repetition. 

Finding an invertible representation is typically a real 
challenge, as the choice is heavily influenced by the ability 
of each feature to convey information about specific prop¬ 


erties of the object, as well as the existing types of objects 
and their distinguishing characteristics. As a second best 
alternative, one should aim at obtaining a as small as 
possible set of features, capable of providing enough infor¬ 
mation so as to solve a given pattern recognition problem 
with suitable performance. 

Previous knowledge and familiarization with several 
types of features, the existing objects to be classified, as 
well as with a variety of pattern recognition methods rep¬ 
resent some of the possible methodologies that can be 
used to select a suitable set of features. It is also pos¬ 
sible to consider multivariate statistical methods such as 
principal component analysis - PC A (e.g. m as a means 
to reduce the number of features through elimination of 
correlations between features, as well as automate feature 
selection methods. 

All in all, the mapping of a single pattern into re¬ 
spective measurements allows this entity to be known in 
an objective, quantitative way, so that eventually respec¬ 
tive models can be derived that not only help explaining 
how the entity behaves, but also allow predictions to be 
made about it. Though these achievements exhibit a high 
level of cognition, we still lack knowledge about how the 
mapped object is related to other co-existing entities. De¬ 
riving these relationships is a task well within the pattern 
recognition realm. 


3 Pattern Recognition 

Once an object has been mapped into respective feature 
vectores /, it occupies a specific position in the associated 
feature space. More specifically, each object is mapped 
into the respective point given by vector / in the feature 
space. Because the components of these vectors corre¬ 
spond to features, which are measurements of the object 
properties, two similar objects tend to be mapped into two 
corresponding vectors that are also similar, implying the 
respective points in the feature space to be relatively near 
one another. The proximity between these two points can 
be quantified in terms of some distance measurement (ob¬ 
serve that there is a virtually infinite number of possible 
distance definitions), such as the Euclidean distance be¬ 
tween the M —dimensional respective feature vectors /i 
and / 2 , i.e. 


^1,2 — 


\ 


M 




(i) 


One important point that needs to be taken into ac¬ 
count is that the concept of distance between patterns is 
a relative issue. Indeed, in case we have only two patterns, 
in the absence of some preliminary imposed criterion, the 
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Euclidean distance between them cannot be taken as be¬ 
ing near or far. This can be realized by considering that 
we lack a reference for comparison, and that even the 
choice of the system of units can be arbitrary (e.g. inches 
or meters, etc.). 

We need at least a third pattern, so that three distances 
can be now calculated, namely 2 , ^ 1,3 and £ 2 , 3 . The 
smallest of these will correspond to the most similar pair 
of objects, and also to derive proportions of similarity by 
considering ratios between the above distances. 

Almost invariably, the entities of a same category will 
exhibit variations intrinsic to their conception (e.g. ge¬ 
netic content) as well as their history. For instance, a 
clam may have lived under harsher conditions, implying 
wearing of its shell. When such entities are transformed 
into respective feature vectors fi and mapped into the re¬ 
spective feature space, they tend to give rise to a cluster 
of points. Figure [3] illustrates this tendency with respect 
to a hypothetical set of clam shells. 



Figure 3: A set of hypothetical clam shells mapped, through some 
respective measurements fi and / 2 , into a corresponding two di¬ 
mensional feature space. The fact that the clams are similar but 
not identical implies in the formation of a cluster of points pos¬ 
sessing a respective spatial dispersion. Total superimposition into a 
single point would have been obtained in case all clams were iden¬ 
tical. Observe that the choice of other features would very likely 
yield different distribution of points in the feature space. 

For analogous reasons, objects that are substantially 
different one another will tend to be mapped into distant 
respective points in the feature space. Now, if we have two 
groups of objects so that all objects in a same group are 
similar one another while objects from distinct groups are 
substantially different, we will obtain two well-separated 
clusters, corresponding to two respective distinct cate¬ 
gories. Figure [4] illustrates this tendency with respect 
to hypothetical groups of clam shells and sea snails. 

The simple principle described above lies at the very 



Figure 4: When two groups of individuals (clam shells and sea 
snails) that are similar within each group, but different between the 
groups are mapped into a two-dimensional feature space, two re¬ 
spective well-defined clusters appear, defining respective categories. 


core of pattern recognition, which can therefore be under¬ 
stood as the endeavor of finding and separating clusters in 
feature spaces and assigning categories to given objects. 

As with the task of deriving effective object represen¬ 
tations, identifying cluster corresponds to another chal¬ 
lenge, especially in case the involved clusters are not well- 
separated, which often turns happen. In addition, clusters 
can appear in several shapes and sizes, which further com¬ 
plicates their identification and separation. These difficul¬ 
ties have motivated the development of a large number of 
pattern recognition methods. 

Generally speaking, categories can be more effectively 
identified when the individuals within each group are 
markedly similar, implying small intra-cluster scattering , 
while exhibiting strongly distinct properties between the 
groups, accounting for large inter-cluster scattering. Oth¬ 
erwise, there could be overlapping of points, i.e. individ¬ 
uals of one group could be found inside other groups. 

We conclude this section by observing that the obten- 
tion of well-separated clusters, and therefore categories, 
in feature spaces depends critically on: ( 1 ) the entities 
of interest initially having intrinsic differentiating char¬ 
acteristics; ( 2 ) deriving pattern representations incorpo¬ 
rating measurements that capture these variations; and 
(3) choosing effective features. Point (1) is particularly 
important because, in case the groups of patterns do not 
exhibit any reasonable intrinsic differences, objective as¬ 
signment of categories is from the scratch doomed to fail¬ 
ure. 
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4 Supervised Pattern Recognition 

Though supervised pattern recognition can only be per¬ 
formed in case the respective categories have already been 
defined, we cover this type of pattern recognition first 
because it is generally simpler and easier than the non- 
supervised counterpart. 

As implied by its own name, supervised patter recogni¬ 
tion, or supervised classification, involves some previous 
knowledge about the groups and their properties. This 
knowledge can be of various types. For instance, we can 
have access to several objects whose category is know, 
so that they can be used in a training stage. Or, we 
can have prototypes of each class, and perhaps also know 
the intra-class variation of each group. Or yet, we may 
know statistical properties of each group. The basic idea 
is, starting from such prior information, to devise some 
means of identifying the category of new entities. So, 
supervised pattern recognition typically involves two sub¬ 
sequent stages: (a) training; and (b) recognition of new 
objects. 

Among the large number of supervised approaches that 
have been developed (e.g. i), for simplicity’s sake we only 
cover one that, despite being very simple, can often lead 
to good performance. We are speaking of the nearest- 
neighbor method (e.g. HD- Given a set of objects with 
known respective categories, they can be mapped into fea¬ 
ture vectors as previously discussed, defining a reference 
population of points in the feature space. As an example, 
we can consider the case in Figure [4j where hypothetical 
sets of clam shells sea snails were mapped into a t two- 
dimensional space by using some arbitrary measurements. 
We repeat that distribution of points in Figure [5] for the 
sake of convenience. 

Now, given a new object of unknown category, we map 
it into the same feature space (e.g. the green or blue 
stars in Fig. |5|, determine the k nearest points and take 
as the most likely category that having majority among 
the k —nearest known objects. Let’s henceforth consider 
k = 1. In the case of the object mapped into the green 
star, we have that the nearest neighbor is a clam, so we 
take this as its most likely category. Observe that the 
same category would have been obtained even for larger 
values of k. Now, in the case of the object mapped into 
the blue star, it is sightly closer to a sea snail, which would 
be the result for k = 1. However, consideration of larger 
values of k would imply instability of this result. 

This method extends immediately to larger number of 
groups and higher dimensional feature spaces. Also, when 
a large set of examples are provided during training, this 
method tends to minimize the chances of getting wrong 
classification results. 



Figure 5: Two hypothetical groups of sea snails and clam shells have 
been mapped into a two-dimensional feature space during the train¬ 
ing stage of the nearest-neighbor method. A new object of unknown 
category mapped into the green star can be understood as being a 
clam for several values of k. However, the most likely category for 
the object mapped into the blue star tends to be unstable. 

5 A Brief Digression 

Though the above approach to clustering involving com¬ 
pact groups that are far away one another corresponds to 
the most often adopted approach, we would like to briefly 
discuss a complementary perspective (e.g. 0 ). Consider 
the situation depicted in Figure [6] 



Figure 6: A situation in which two groups give rise to a single 
cluster, but in such a way that they can still be properly separated, 
e.g. by considering the linear decision boundary shown in magenta. 

In this hypothetical case, we have two species of clams: 
one characterized by being taller than wider, and the 
other vice-versa. The individuals from each species are 
also assumed to appear with different diameters (i.e. the 
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largest distance between any possible pair of its points). 
In case we choose /i to correspond to the ratio between 
the width and height of the shells, and to coincide with 
the their diameters, the distribution of points in the fea¬ 
ture space could be similar to that shown in Figure [6j 

Interestingly, though none of the two species give rise 
to a cluster, characterized by a compact, well-separated 
group of points, the two groups nevertheless resulted or¬ 
ganized and separated in the feature space (no overlaps). 
Indeed, it is possible to perfectly separate the two groups 
by considering the linear boundary (also called separatrix ) 
shown in magenta in Figure |6j and we can say that the 
two groups are linearly separable, as indeed also are the 
groups in Figure |4j However, we will not refer to these 
separable groups as clusters. 

It is interesting to compare the two situations shown 
in Figure [4] and [6] Though the two groups are linearly 
separable in both cases, in case we did not know a priori 
that there are two groups, we would not be able to find 
them from the situation in Figure [6j while the separation 
would be obvious in the other case. On the other hand, 
the situation in Figure [6] has the potential for allowing en¬ 
hanced resolution and discriminability between the points 
in each group, as a consequence of each group occupying a 
relatively larger area than it would be otherwise obtained 
in the case of two compact and distant groups. 

So, in case we have a priori indication about the prop¬ 
erties of the categories, as it happens in the training stage 
of supervised learning, it is possible to sacrifice the sep¬ 
arability of the groups for enhanced resolution and char¬ 
acterization of objects inside each group. Observe that 
this approach would reduces the margin of error in the 
objects mapping, and could therefore result in overlap. 
In summary, the above discussion emphasizes that, given 
the limited resolution of measurements and feature space 
representations (especially in the case of discrete measure¬ 
ments), there exists a trade-off between margin of classi¬ 
fication error and enhanced resolution. The decision of 
which should take priority depends critically on each spe¬ 
cific problem and should be exerted with special care. 

6 Unsupervised Pattern Recogni¬ 
tion 

Now, we proceed to the more challenging problem of un¬ 
supervised classification, sometimes also called clustering : 
nothing is known about the categories or their statistics 
properties, and we only have access to a set of observed 
feature vectors defining respective distribution of points 
in a feature space defined by the chosen measurements. 
An example can be immediately derived from Figure [4] 
by disregarding the respective known categories, giving 


rise to the distribution of undistinguishably marked points 
shown in Figure [7] 



Figure 7: A set of points (orange discs) are to be separated into 
two categories by using the k— means algorithm. Two respective 
seeds are randomly assigned (e.g. the two empty stars), and the 
points closer to each of them are successively identified, while their 
centers of mass are then taken as new corresponding seeds. The 
method is terminated when the positions of the seeds become stable. 
A properly converging result is shown in terms of the two seeds 
represented as the blue and green stars, which define a respective 
Voronoi partitioning of the feature space, so that the border between 
the two Voronoi cells constitues a linear separatrix. 

Though belonging to the ‘easy’ type of unsupervised 
classification, we can use this situation to introduce one of 
the simplest and best known clustering methods, namely 
A;—means (e.g. ED- 

First, we have to choose how many categories will be 
taken into account. In this case, the obvious choice is 
k = 2, but other situations involving less separated groups 
can be more challenging. Then, we need two respective 
‘seeds 7 , around which the respective categories can be or¬ 
ganized. These can be either drawn randomly, or we may 
have some intuition or insight about the possible location 
of the clusters. 

The k —means algorithm then identifies which of the 
feature space points are closer to each of the seeds, defin¬ 
ing two preliminaries partitions of the feature space. The 
centers of mass of the points inside each of these parti¬ 
tions are then calculated and understood as new respec¬ 
tive seeds. The method proceeds until the position of the 
seeds become stable, which tends to happen in just a few 
steps. Hopefully, the resulting seeds will correspond to the 
sough categories, spanning respective regions so that each 
point inside these regions are closer to the corresponding 
seed, defining a Voronoi partitioning of the feature space. 
Figure [7] illustrates the two obtained categories. Observe 
that the Voronoi partition in this case presents a linear 
separation border. 
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Though relatively simple, this method often provides 
good performance. The main difficulty is that the posi¬ 
tions of the original seeds can have a strong effect on the 
final result. For instance, two seeds may converge to the 
same cluster. One possibility to improve the result is to 
perform the algorithm several times with respect to many 
initial seeds positions, and to disregard the cases resulting 
in seeds superimposition. Another limitation is that the 
number of categories needs to be properly estimated. 


7 Concluding Remarks 

As a means of understanding and communicating about 
the world, human beings have mapped the several of its 
entities into abstract concepts or categories, which ulti¬ 
mately played an important role on the development of 
language and science. In the present work, we discussed 
how the task of pattern recognition has been fundamental 
in this process. 

We have seen that pattern recognition typically involves 
two subsequent stages. First, each of the entities to be rec¬ 
ognized is transformed into a quantitative representation 
in terms of a feature vector, a stage that we called pat¬ 
tern cognition. Then, the feature vectors corresponding 
to several objects are analyzed in order to identify their 
respective categories, constituting the patter recognition 
proper. This can proceed in two main manners, one pro¬ 
ceeding from scratch, without any information about the 
possible categories or their statistics (unsupervised pat¬ 
tern recognition), while the other is based on a training 
stage involving samples or prototypes with known cate¬ 
gories. We have described two simple but relatively ef¬ 
fective respective methods, namely A;—neighbors (super¬ 
vised) and A:—means (unsupervised). 

The reader should keep in mind that, as a consequence 
of the central importance of pattern recognition in sci¬ 
ence and technology, many other interesting and effective 
concepts and methods exist, including the application of 
principal component analysis (e.g. 0) as a means to re¬ 
duce the dimensionality of the feature space, as well as 
a large number of alternative interesting methods for su¬ 
pervised and unsupervised pattern recognition. 
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Costa’s Didactic Texts - CDTs 


CDTs intend to be a halfway point between a 
formal scientific article and a dissemination text 
in the sense that they: (i) explain and illustrate 
concepts in a more informal, graphical and acces¬ 
sible way than the typical scientific article; and 
(ii) provide more in-depth mathematical develop¬ 
ments than a more traditional dissemination work. 

It is hoped that CDTs can also integrate new 
insights and analogies concerning the reported 
concepts and methods. We hope these character¬ 
istics will contribute to making CDTs interesting 
both to beginners as well as to more senior 
researchers. 

Each CDT focuses on a limited set of interrelated 
concepts. Though attempting to be relatively 
self-contained, CDTs also aim at being relatively 
short. Links to related material are provided in 
order to complement the covered subjects. 

Observe that CDTs, which come with absolutely 
no warranty, are non distributable and for non¬ 
commercial use only. 

The complete set of CDTs can be found 
at: https://www.researchgate.net/project/ 
Costas-Didactic-Texts-CDTs. 





